If Web metrics has always seemed like an overly complicated endeavor that left you scratching your head, don’t feel singled out. The Web is still a baby, changing so fast that we’re still learning how to measure it. But if all the newfangled metrics concepts and methodologies leave you wanting for some simplicity, read on. The Unified Field Theory of Web Metrics attempts to condense all Web activity into three concepts, and it’s the subject of this month’s Biznology newsletter.
“You need to check your open rate and your bounce rate,” advises an e-mail marketing expert. “Subscribers and comments tell you the most about how well your blog is doing,” opines another wizard. “Until something better comes along, podcasts are best measured by download counts,” a third guru offers.
Now, if this advice was wrong, it would be easy to tell you to ignore it. The problem is that all the advice is good, as far as it goes. As with any new area, Web marketing tactics are emerging (and changing) faster than anyone can figure out how to measure their effectiveness. And each new tactic seems to drag along its own measurement language, which can make things more precise for an expert, but leave ordinary folks confused.
If Web metrics makes your brain hurt, don’t worry. We can simplify things for clarity. If we end up oversimplifying things a bit, that’s the price you pay. As mathematician John von Neuman observed, “There’s no sense in being precise when you don’t even know what you’re talking about.”
Just as Einstein and others pursued the Unified Field Theory of Physics, we need to think about a Unified Field Theory of Web Metrics. (And when they give a Nobel Price for Web Analytics, I’m all set.) All of these competing metrics are just too confusing. Each of them has its place, but it relegates analysis to the realm of the expert instead of the practitioner. Let’s look at metrics in a unified way across all marketing tactics so that we could easily analyze anything. Remember, every day new marketing tactics show up—the Web doesn’t stand still. We need some organizing principles that help us think about any tactic in the same way.
To do that, we need to think about the basics of Web activity metrics:
- Did they see it? The first thing we want to know is if our content was viewed by the customer. Was the ad shown? Was the e-mail seen? Was the blog post viewed? We call this measurement the impression.
- Did they choose it? If seen, was it acted upon? Did they click on our message to go deeper? This one is called the selection. (Because AJAX user interfaces allow selections without clicks, we shouldn’t call this a clickthrough, although usually selection is done by clicking.)
- Did they do it? What did we want the customer to do? Buy something? Fill out a contact form? Call our toll-free number? This metric is called the conversion.
Now, it turns out that this isn’t all so breathtakingly new. We’ve talked in this space in the past about identifying your Web site’s goals (its conversions). Most Web metrics software allows you to identify any Web event as a conversion, so that you can track it. In addition, selections (usually called clickthroughs) are relatively easy to count. Metrics software can track any click made on your page, and referral tracking can track clicks made elsewhere to bring people to your site.
Now for the bad news. Impressions aren’t so easy to count sometimes.
We’ll start with the simplest case: banner ads. Banner ads are simple because they call their impressions, uh, impressions. So when your metrics reports list the number of impressions for a banner ad, it means just what you think it does—that’s how many times it was shown. And, happily, contextual ads and paid search ads also call their impressions by that name. So they are simple, too.
Web pages are only a bit more complicated. First off, Web page impressions are usually referred to as page views, even though it has the same meaning that impression does for banners and other ads. One page view means that the page was shown once, so you can think of it as a page impression. Your metrics software probably uses a Web beacon(sometimes called single-pixel tracking) technique that is more accurate than the old log file approaches of the past. (Unfortunately, what you really want to count are content impressions—exactly which ads showed up on that personalized page, for example—so metrics systems are scrambling to count content on pages rather than the pages themselves. Still, we really want to count impressions, just as with everything else.)
It gets even tougher. Let’s take Web feeds, such as RSS, that are used whenever someone subscribes to a blog. Your customer is using a reader (sometimes called a blog reader or RSS reader or feed reader), such as Bloglines, to look at all subscribed blogs. When a particular blog is selected, your customer can read the individual blog posts from within the reader.
The problem is that no metrics can be easily gathered from within blog readers, so no one knows for sure how many posts are actually read. You can take a guess that every subscriber sees each post, but that surely overstates the number of impressions. Metrics experts are working on adapting the Web beacon technique to register a page view when blog posts are displayed within the blog reader, but until then we have to estimate based on subscribers.
Some metrics experts suggest that you send short excerpts of your blog posts in your blog feed, rather than the full posts. Doing so makes it easy to count impressions of the full post, because customers click through to your Web site to read it, thus making it as easy to count as any other Web page view. But many usability experts decry this practice because customers prefer to read everything in their feed readers, without bouncing back and forth to your Web site. Don’t annoy your customers just to solve a measurement problem—just accept that blog feed impression counts are not terribly precise. Until something better comes along, we can make the simplifying assumption that the impressions of your blog posts in blog readers are equal to your number of subscribers.
But that begs the question of how many subscribers you have. Most blogs and other feeds allow the subscriber to choose the format of the feed (providing buttons for XML, RSS, or Atom, for example), which makes it harder for the feed owner to count the number of subscribers. Feed aggregation companies, such as Feedburner, have arisen to solve this problem. Feedburner and its competitors provide a single subscribe button for you to place on your site, with Feedburner providing the best format to each subscriber’s individual feed reader. Because all subscribers go through Feedburner, the feed owner can be provided with statistics to show the number of subscribers, for example.
But it’s not so simple, unfortunately. Yahoo!, AOL, and other companies often aggregate feeds for their subscribers, so that when an AOL user subscribes to a feed, AOL might provide that feed from its own cache of feeds rather than alerting Feedburner. While an efficient way of providing feeds to subscribers, it guarantees under-counting of subscribers—possibly by 25%. However, precision is less important than broad trends, so use whatever feed subscription metrics that you can lay your hands on.
E-mail is even more complicated to measure than blogs, because so many steps happen before your customer clicks through to your Web site. When you send out a group of targeted e-mails for your campaign, you can count how many you sent. But that doesn’t mean that each one was received. To know how many are received, you must monitor how many e-mails bounce—get returned to you as undeliverable. So, the number you send less the number bounced tells you the number delivered.
But all delivered e-mail is not received, unfortunately. E-mail spam filters block many messages from reaching their recipients, even when those messages are not spam. Some are blocked by mail servers; others by spam filters on each individual customer’s personal computer. Usually, spam filters do not return flagged e-mail to the sender, so you’ll never know how many e-mails you sent were never received.
So, you can estimate the number received by subtracting bounced e-mails from sent e-mails, but you know it is an overestimate. You can use that number as the e-mail “in box” impressions—that’s the number of e-mails that were seen in customer in boxes. Some of those e-mails are opened—this is usually referred to as the open rate, but it isn’t any different from a clickthrough rate on any content. If you deliver a lot of e-mails but few of them are opened, you may not have the best e-mail subject line. (If you’re wondering how anyone can measure how many e-mails are opened, remember that e-mail can be sent in the form of a Web page and the Web beacon technique can be applied to it as long as the customer is connected to the Internet when opening the mail.)
So, e-mail is a bit complicated. You can measure two impressions (seeing the subject line in the in box and seeing the entire mail message) and neither of them are called impressions by the experts. And the measurements themselves are not the most precise either.
Measuring podcasts is not that simple either. Podcasts must be downloaded from a Web site and listened to on an iPod or a personal computer. Your metrics expert can tell you if your podcast was downloaded, but not whether anyone ever listened to it. Metrics gurus are experimenting with using voluntary surveys of podcast listeners that install software to track what they listen to (similar to how television ratings are measured), but it’s not clear how widespread or accurate such an approach will become. Until we get something better, we might as well equate downloads with impressions.
But the toughest impression to measure is the organic search impression. Search engines display the title of your page in the search results, along with a “snippet,” made up of text extracted from your page that contains the searcher’s keyword. But how do you know when Google is showing your page in the results?
It’s not that easy to tell. Yahoo! will tell you how many searches it performed for popular keywords in the last month, using its keyword tool, but Google won’t. (You could multiply the Yahoo! number by two to guess the Google number, because most estimates are that it has twice the market share.) Once you get the number of searches, you could decide to count an impression for any result on the first page of search results (the top ten). But this is a very wild guess, and search results change frequently, so it’s not very easy to estimate organic search impressions.
Most experts track search rankings instead of impressions, but you can see that it is possible to at least estimate impressions if you want to. Even simpler might be to treat every search as a potential impression, because whether your page was not shown or not clicked, it’s still a missed opportunity that you’d want to address.
In the future, when search is personalized so that each searcher gets results tailored just for them, only the search engines themselves will know what search results were displayed. It’s possible that the search engines will sell this information to hungry marketers to settle the question of organic search impressions once and for all.
So, tracking impressions is a wee bit complicated, in part because we use different names for the same thing, but in part because the numbers are not terribly precise yet. So, instead of insisting on precision, why can’t we do our best to estimate impressions wherever we can’t count them directly?
So, for banner ads, we count them directly and that’s great, but why can’t we estimate e-mail inbox impressions? Has a study been done that shows that approximately 10% of legitimate marketing e-mail gets tangled in spam filters? Then adjust all your estimated impressions by 10%. Could we estimate organic search impressions based on your ranking? If your page lands #1-#10, then your impressions equal the number of searches. If your ranking lands on the second page, then we estimate that about 15% of searchers move to the second page, so your impressions can be estimated at the number of searches multiplied by 15%. Do we similarly have any studies that show the percentage of blog posts actually seen in feed readers? Do we know how many subscriptions are processed by aggregators? We could apply the same approach to adjust subscriber counts to impression counts while we wait for a Web beacon solution.
If we unify all the different fields that we currently track, we can use estimation to make our numbers somewhat more precise. Right now, the numbers are imprecise but we make up for that by confusing everyone to death about them. My advice is that we simplify how we think about these numbers into impressions, selections, and conversions. And if impressions turns out to be a relatively imprecise term for some of what we want to count, let’s do the best we can and estimate the rest. As Jim Sterne says, “Web analytics are not precise, but they are true.”
Use the numbers you have to make decisions and you’ll be better off than if you have no numbers. And simplify the numbers until you understand them. Web metrics experts may accuse me of oversimplifying, but you’re better off understanding what you are doing than being a slave to accuracy. After all, even with inaccurate measurements, you can still analyze trends and make good decisions. If we can reach relatively accurate estimates that normal business people can understand, then those normal people can make decisions based on metrics instead of relying on metrics gurus. I think that would be a good thing.
